<html>
<head><meta charset="utf-8"><title>Use in-memory index instead of writing to disk? · t-cargo · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/index.html">t-cargo</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Use.20in-memory.20index.20instead.20of.20writing.20to.20disk.3F.html">Use in-memory index instead of writing to disk?</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="222193170"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Use%20in-memory%20index%20instead%20of%20writing%20to%20disk%3F/near/222193170" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Use.20in-memory.20index.20instead.20of.20writing.20to.20disk.3F.html#222193170">(Jan 09 2021 at 22:36)</a>:</h4>
<p>Is there a way to tell cargo to store the <a href="http://crates.io">crates.io</a> index in memory instead of on disk? I've been staring at <code>Updating crates.io index</code> for about 15 minutes now :/</p>



<a name="222193179"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Use%20in-memory%20index%20instead%20of%20writing%20to%20disk%3F/near/222193179" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Use.20in-memory.20index.20instead.20of.20writing.20to.20disk.3F.html#222193179">(Jan 09 2021 at 22:37)</a>:</h4>
<p>hmm, maybe I could play games with CARGO_HOME</p>



<a name="222193199"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Use%20in-memory%20index%20instead%20of%20writing%20to%20disk%3F/near/222193199" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jon Gjengset <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Use.20in-memory.20index.20instead.20of.20writing.20to.20disk.3F.html#222193199">(Jan 09 2021 at 22:38)</a>:</h4>
<p>You're going to love <a href="https://github.com/rust-lang/cargo/pull/8890">https://github.com/rust-lang/cargo/pull/8890</a> when it one day lands :p</p>



<a name="222193237"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Use%20in-memory%20index%20instead%20of%20writing%20to%20disk%3F/near/222193237" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Use.20in-memory.20index.20instead.20of.20writing.20to.20disk.3F.html#222193237">(Jan 09 2021 at 22:38)</a>:</h4>
<p>indeed I am <span aria-label="laughing" class="emoji emoji-1f606" role="img" title="laughing">:laughing:</span></p>



<a name="222193243"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Use%20in-memory%20index%20instead%20of%20writing%20to%20disk%3F/near/222193243" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Use.20in-memory.20index.20instead.20of.20writing.20to.20disk.3F.html#222193243">(Jan 09 2021 at 22:38)</a>:</h4>
<p>that lets cargo only download the crates it cares about, instead of metadata for all of <a href="http://crates.io">crates.io</a>?</p>



<a name="222193250"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Use%20in-memory%20index%20instead%20of%20writing%20to%20disk%3F/near/222193250" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jon Gjengset <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Use.20in-memory.20index.20instead.20of.20writing.20to.20disk.3F.html#222193250">(Jan 09 2021 at 22:39)</a>:</h4>
<p>Exactly!</p>



<a name="222193254"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Use%20in-memory%20index%20instead%20of%20writing%20to%20disk%3F/near/222193254" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Use.20in-memory.20index.20instead.20of.20writing.20to.20disk.3F.html#222193254">(Jan 09 2021 at 22:39)</a>:</h4>
<p>that would be really nice</p>



<a name="222193261"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Use%20in-memory%20index%20instead%20of%20writing%20to%20disk%3F/near/222193261" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Use.20in-memory.20index.20instead.20of.20writing.20to.20disk.3F.html#222193261">(Jan 09 2021 at 22:39)</a>:</h4>
<p>right now cargo probably takes more time than the build itself</p>



<a name="222193315"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Use%20in-memory%20index%20instead%20of%20writing%20to%20disk%3F/near/222193315" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jon Gjengset <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Use.20in-memory.20index.20instead.20of.20writing.20to.20disk.3F.html#222193315">(Jan 09 2021 at 22:40)</a>:</h4>
<p>See in particular the graphs in <a href="https://github.com/rust-lang/cargo/pull/8890#issuecomment-737610713">https://github.com/rust-lang/cargo/pull/8890#issuecomment-737610713</a></p>



<a name="222193316"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Use%20in-memory%20index%20instead%20of%20writing%20to%20disk%3F/near/222193316" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jon Gjengset <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Use.20in-memory.20index.20instead.20of.20writing.20to.20disk.3F.html#222193316">(Jan 09 2021 at 22:40)</a>:</h4>
<p>And then the ones in <a href="https://github.com/rust-lang/cargo/pull/8890#issuecomment-738316828">https://github.com/rust-lang/cargo/pull/8890#issuecomment-738316828</a></p>



<a name="222193328"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Use%20in-memory%20index%20instead%20of%20writing%20to%20disk%3F/near/222193328" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Use.20in-memory.20index.20instead.20of.20writing.20to.20disk.3F.html#222193328">(Jan 09 2021 at 22:41)</a>:</h4>
<p><span class="user-mention" data-user-id="120054">@Jon Gjengset</span> how fast is the disk speed in those graphs?</p>



<a name="222193388"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Use%20in-memory%20index%20instead%20of%20writing%20to%20disk%3F/near/222193388" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jon Gjengset <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Use.20in-memory.20index.20instead.20of.20writing.20to.20disk.3F.html#222193388">(Jan 09 2021 at 22:42)</a>:</h4>
<p>That was against an SSD, but given how _tiny_ the files you download are, my guess is disks speed is almost irrelevant. I say that in part because I found that bandwidth was basically irrelevant to how well the HTTP API performed</p>



<a name="222193392"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Use%20in-memory%20index%20instead%20of%20writing%20to%20disk%3F/near/222193392" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Use.20in-memory.20index.20instead.20of.20writing.20to.20disk.3F.html#222193392">(Jan 09 2021 at 22:42)</a>:</h4>
<p>well there's a <em>lot</em> of different files</p>



<a name="222193393"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Use%20in-memory%20index%20instead%20of%20writing%20to%20disk%3F/near/222193393" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Use.20in-memory.20index.20instead.20of.20writing.20to.20disk.3F.html#222193393">(Jan 09 2021 at 22:42)</a>:</h4>
<p>at that point filesystem metadata is significant</p>



<a name="222193397"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Use%20in-memory%20index%20instead%20of%20writing%20to%20disk%3F/near/222193397" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Use.20in-memory.20index.20instead.20of.20writing.20to.20disk.3F.html#222193397">(Jan 09 2021 at 22:43)</a>:</h4>
<p>(I have an extremely slow disk on the machine that's taking 15 minutes)</p>



<a name="222193401"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Use%20in-memory%20index%20instead%20of%20writing%20to%20disk%3F/near/222193401" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jon Gjengset <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Use.20in-memory.20index.20instead.20of.20writing.20to.20disk.3F.html#222193401">(Jan 09 2021 at 22:43)</a>:</h4>
<p>When you download the entire index, yes, but not if you download only the metadata you need. Then we're talking usually a handful of files</p>



<a name="222193404"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Use%20in-memory%20index%20instead%20of%20writing%20to%20disk%3F/near/222193404" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jon Gjengset <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Use.20in-memory.20index.20instead.20of.20writing.20to.20disk.3F.html#222193404">(Jan 09 2021 at 22:43)</a>:</h4>
<p>Depending on the size of your dependency closure</p>



<a name="222193447"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Use%20in-memory%20index%20instead%20of%20writing%20to%20disk%3F/near/222193447" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Use.20in-memory.20index.20instead.20of.20writing.20to.20disk.3F.html#222193447">(Jan 09 2021 at 22:44)</a>:</h4>
<p>well mostly I want to compare it to the git version</p>



<a name="222193456"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Use%20in-memory%20index%20instead%20of%20writing%20to%20disk%3F/near/222193456" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jon Gjengset <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Use.20in-memory.20index.20instead.20of.20writing.20to.20disk.3F.html#222193456">(Jan 09 2021 at 22:44)</a>:</h4>
<p>Yeah, the git version downloads the entire index. Worse, actually, it downloads the entire _history_ of the entire index.</p>



<a name="222193460"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Use%20in-memory%20index%20instead%20of%20writing%20to%20disk%3F/near/222193460" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jon Gjengset <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Use.20in-memory.20index.20instead.20of.20writing.20to.20disk.3F.html#222193460">(Jan 09 2021 at 22:44)</a>:</h4>
<p>The http version downloads only the current version of the subset of the index you're using</p>



<a name="222193461"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Use%20in-memory%20index%20instead%20of%20writing%20to%20disk%3F/near/222193461" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jon Gjengset <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Use.20in-memory.20index.20instead.20of.20writing.20to.20disk.3F.html#222193461">(Jan 09 2021 at 22:44)</a>:</h4>
<p>Which is a small number of small files</p>



<a name="222193476"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Use%20in-memory%20index%20instead%20of%20writing%20to%20disk%3F/near/222193476" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Steven Fackler <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Use.20in-memory.20index.20instead.20of.20writing.20to.20disk.3F.html#222193476">(Jan 09 2021 at 22:45)</a>:</h4>
<p>to answer specifically your initial question, you can point CARGO_HOME to a tmpfs which should live entirely in memory</p>



<a name="222193477"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Use%20in-memory%20index%20instead%20of%20writing%20to%20disk%3F/near/222193477" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jon Gjengset <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Use.20in-memory.20index.20instead.20of.20writing.20to.20disk.3F.html#222193477">(Jan 09 2021 at 22:45)</a>:</h4>
<p>The incremental index update is somewhat better, depending on how out-of-date you are. But even then you're downloading the history since last you updated. With the HTTP one, you're not even doing disk writes for any index files that haven't changed.</p>



<a name="222193479"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Use%20in-memory%20index%20instead%20of%20writing%20to%20disk%3F/near/222193479" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jon Gjengset <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Use.20in-memory.20index.20instead.20of.20writing.20to.20disk.3F.html#222193479">(Jan 09 2021 at 22:45)</a>:</h4>
<p>Oh, yeah, sorry for derailing <span aria-label="sweat smile" class="emoji emoji-1f605" role="img" title="sweat smile">:sweat_smile:</span></p>



<a name="222193652"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Use%20in-memory%20index%20instead%20of%20writing%20to%20disk%3F/near/222193652" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Steven Fackler <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Use.20in-memory.20index.20instead.20of.20writing.20to.20disk.3F.html#222193652">(Jan 09 2021 at 22:50)</a>:</h4>
<p>we do periodically truncate the index's history but it's still downloading info for all crates with some amount of history</p>



<a name="222193719"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Use%20in-memory%20index%20instead%20of%20writing%20to%20disk%3F/near/222193719" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jon Gjengset <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Use.20in-memory.20index.20instead.20of.20writing.20to.20disk.3F.html#222193719">(Jan 09 2021 at 22:52)</a>:</h4>
<p>Huh, interesting, I had no idea you truncated the history of that repository! A rebase squash with a forced update on the client or something like that?</p>



<a name="222193946"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Use%20in-memory%20index%20instead%20of%20writing%20to%20disk%3F/near/222193946" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Steven Fackler <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Use.20in-memory.20index.20instead.20of.20writing.20to.20disk.3F.html#222193946">(Jan 09 2021 at 23:00)</a>:</h4>
<p>yeah I think it basically just involves squashing the whole history of the repo down to 1 commit and force pushing that up</p>



<a name="222193969"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Use%20in-memory%20index%20instead%20of%20writing%20to%20disk%3F/near/222193969" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Steven Fackler <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Use.20in-memory.20index.20instead.20of.20writing.20to.20disk.3F.html#222193969">(Jan 09 2021 at 23:00)</a>:</h4>
<p>with publishing temporarily turned off obv</p>



<a name="222198442"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Use%20in-memory%20index%20instead%20of%20writing%20to%20disk%3F/near/222198442" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eh2406 <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Use.20in-memory.20index.20instead.20of.20writing.20to.20disk.3F.html#222198442">(Jan 10 2021 at 01:07)</a>:</h4>
<p><span class="user-mention silent" data-user-id="120054">Jon Gjengset</span> <a href="#narrow/stream/246057-t-cargo/topic/Use.20in-memory.20index.20instead.20of.20writing.20to.20disk.3F/near/222193719">said</a>:</p>
<blockquote>
<p>Huh, interesting, I had no idea you truncated the history of that repository! A rebase squash with a forced update on the client or something like that?</p>
</blockquote>
<p>see <a href="https://github.com/rust-lang/crates-io-cargo-teams/issues/47">https://github.com/rust-lang/crates-io-cargo-teams/issues/47</a> for all the gory detaels</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>